Public Engagement 


How to... 


...@Valuate your public 
engagement support 
programme 


Guidance on how to evaluate the impact of your public 
engagement support programme and to monitor ‘culture 
change’ in your institution 


Evaluating your work is key to reflecting on what is working well and where improvements can be 
made, as well as assessing the impact of your work. The key steps to any evaluation plan, whether 
you are evaluating a public engagement activity, a culture change project, or a long term support 
programme, are the same. 


The NCCPE website provides resources to help with the basic steps to evaluating your work. 


In this guide we share some of the ways people have used evaluation to measure the impact of 
their work to support engagement effectively. 


Logic models 


A really helpful tool for planning your approach is to use a logic model. Logic models are a tool 
used by many funders, managers and evaluators of complex interventions to help them plan and 
evaluate their success. Using a logic model enables you to map your project, considering what you 
are hoping to achieve, and how you plan to achieve this, and to make your assumptions about 
change explicit. A typical logic model will include the following features: 


e Current situation - A description of the situation you are trying to change 
e Aims -— what you hope to achieve 

e Activities — what you are going to do to achieve the aims 

e Outputs — what you plan to create 

e Outcomes — what you expect to happen as a result 

e Impacts — what is the long term effect you hope to achieve 

e Assumptions — that you are making in designing your approach 

e External factors — that could influence the outcomes of your project 


A logic model can provide a useful framework to map out these things — and understand better 
the shape of what you are trying to do. Working through the logic model with those who will be 
involved in the project (e.g. team members, partner organisations) helps you have a useful 
discussion about your project, and highlights the assumptions you are making. It helps you make 
explicit how you think the activities you are planning will lead to the desired impacts. 


A logic model can be used to inform your approach to evaluation. What questions do you have 
about your approach? What do you want to know? It may be that you are interested in the current 
situation, if and how the activities influence the outputs and outcomes, or whether you have 
actually made a difference. Your questions might focus on the current situation; the processes you 
are using; or the outcomes and impacts. A logic model helps you make those important decisions 
about where to focus your attention. 


What are outputs, outcomes and impacts? 


When planning an evaluation it is helpful to differentiate between outputs, outcomes and impact 
as these provide useful ways to define the different ways in which your work can contribute to 
change, over time. 


Outputs are usually tangible products, and as such are relatively easy to capture. Examples of 
outputs for a support programme might include: 


e Public engagement award scheme operational 

e Promotion criteria for all grades now include public engagement 

e Performance review paper work references public engagement 

e A published definition of public engagement 

e Integration of public engagement within strategic plans 

e Changes to planning cycles that support public engagement — e.g. included in workload 
planning 

e Changes to formal governance arrangements to ensure oversight of public engagement 

e Changed senior manager job descriptions that include public engagement 

e Evaluation plan 

e Breadth and depth of participation in activities. — e.g. numbers of staff trained and/or 
advised; attending events; subscribing to newsletters or joining networks; roles of 
champions/ coordinators created 


Monitoring outputs is relatively straightforward. You should make sure you have routine ways to 
collect this data. 


When considering numbers of people involved it is often useful to capture some information 
about them e.g. for staff you might want to capture their role, discipline, or department; for 
members of the public you may want to capture demographics. 


Outcomes and impacts 


Outcomes are usually easier to capture than impacts, as they happen in a quicker time frame. 
Outcomes are the results of the activity and could be described as immediate impacts. Impacts 
relate to longer term change. 


Typical outcomes for a support programme might include: 


e Greater awareness amongst staff about what public engagement means and confidence in 
the agreed definition 

e Staff make use of strategy and definition in informing their PE work 

e Senior staff champion engagement in key meetings, and tackle areas where engagement is 
not supported well 

e The quality of public engagement activity increases, due to effective evaluation and 
support 

e Submission of public engagement with research activities within the RCUK Research 
Outcomes System or ResearchFish 

e Inclusion of public engagement within RCUK Pathways to Impact or other funders’ schemes 

e Improved staff practices and attitudes to public engagement 

e Public engagement more regularly included in institutional meetings, networks, activities 


In thinking about longer term impacts, it can be helpful to categorise these into three types: 


e Conceptual impacts: changes in knowledge, understanding, attitude, or awareness, for 
instance: 
o Arobust, shared understanding of Public Engagement 
o Clear understanding where to go to get support for public engagement 
o Staff recognise the value of public engagement to the university 


e Instrumental impacts: changes to policies, behaviour or practices, for instance: 
o Additional funding is secured to support high quality public engagement with 


research 
o Engaged staff are rewarded for their engagement work alongside their research and 


teaching work 
o More effective engagement leads to mutually beneficial impacts, that are captured 


e Capacity building impacts: development of skills to do engagement well, for instance: 
o Larger number of staff offer leadership for engagement, across all levels of the 


institution 
o Partnerships with key organisations enhance the work of the university, as well as 


delivering value to the partners involved 
o Champions are equipped to support their colleagues to get involved 


Planning your evaluation 


Here we provide some examples of aims and objectives you might set for your support 
programme, and a list of possible outputs, outcomes and impacts for each of these, to help you 


plan your evaluation. 


Your overarching aim might be: To create a culture within the HEI where excellent public 


engagement with research is embedded 


To achieve this aim, you might set three different objectives: 
1. To develop and demonstrate a strategic commitment to public engagement 
2. To reward and recognise staff and students for their public engagement work 
3. To create effective approaches to encourage and support staff at all levels to become 
involved 


Below, we look at each of these objectives in turn and provide suggestions of the potential 
outputs, outcomes and impacts. 


AIM: TO CREATE A CULTURE WITHIN THE HEI WHERE EXCELLENT PUBLIC ENGAGEMENT WITH 


RESEARCH IS EMBEDDED 


OBJECTIVE 1: DEVELOP AND DEMONSTRATE A STRATEGIC COMMITMENT TO PUBLIC 
ENGAGEMENT BY... 


e Developing a shared understanding of public engagement across the institution within 12 
months 

e Embedding public engagement within the strategic goals of the institution; planning cycles; 
formal governance; academic workload planning; and within responsibility of senior managers 

e Creating a public engagement strategy for the institution (and/ or departments) in 
consultation with staff, students and external partners 

e Integration of public engagement into the core activities of the organisation, including 
measuring the quality and quantity of public engagement activities 

e Allocation of strategic funding to support engagement work 

e Intelligence gathering about public perceptions, and into areas of need/ opportunity to inform 
planning 

e Development of effective long term partnerships with key organisations outside of the 
institution 


Potential Outputs Potential Outcomes Potential Impacts 


Shared definition of public Greater awareness amongst staff about More effective engagement 


engagement 

Integration of public engagement 
within strategic plans 

Changes to planning cycles that 


what public engagement means and 
confidence in the agreed definition 


Funding and support in place to deliver 


strategic plans / engagement strategy 


leads to mutually beneficial 
impacts, that are captured 
Additional funding is 
secured to support high 


Staff make use of strategy and definition in 
informing their PE work 

Senior staff champion engagement in key 
meetings, and tackle areas where 
engagement is not supported well 

The quality of public engagement activity 
increases, due to effective evaluation and 
support 

Input prepared for the REF, within HEIF, 
Knowledge Transfer Grants 

Submission of public engagement with 
research activities within the RCUK 
Research Outcomes System or ResearchFish 
Inclusion of public engagement within RCUK 
Pathways to Impact or other funders’ 
schemes 


quality public engagement 
with research 

Larger number of staff offer 
leadership for engagement, 
across all levels of the 
institution 

Partnerships with key 
organisations enhance the 
work of the university, as 
well as delivering value to 
the partners involved 


support public engagement 

— e.g. included in workload planning 
Changes to formal governance 
arrangements to ensure oversight of 
public engagement 

Changed senior manager job 
descriptions that include public 
engagement 

Evaluation plan 

Value of strategic funds to support 
central engagement unit 

Number of external partners 
engaged 

Outputs from the research into 
public perceptions, and areas of 
need/opportunity 

Sign up to NCCPE manifesto 


OBJECTIVE 2: REWARD AND RECOGNISE STAFF AND STUDENTS FOR THEIR PUBLIC 
ENGAGEMENT WORK BY... 


e Developing an annual award for public engagement with research within 12 months 

e Supporting researchers to apply for the NCCPE Engage awards (ongoing) 

e Developing recognition for public engagement across all grades in promotion criteria within 24 
months 

e Including public engagement within performance review processes within 24 months 


Potential Outputs 


Public engagement award 
scheme operational 
Applications for Engage 
Award 

Promotion criteria for all 
grades include public 
engagement 

Performance review paper 
work references public 
engagement 


Raised awareness of the value of 
public engagement amongst staff 
Increased number of applications to 
the universities award scheme, and 
the Engage Awards 

Applicants for promotion choose to 
cite their engagement work within 
their application, and panels choose 
to recognise this in their assessment 
Engagement is raised within 
performance review 


Potential Outcomes Potential Impacts 


Staff recognise the value of public 
engagement to the university, and an 
increased number get involved in public 
engagement as a consequence 

Engaged staff are rewarded for their 
engagement work alongside their research 
and teaching work 


OBJECTIVE 3: CREATE EFFECTIVE APPROACHES TO ENCOURAGE AND SUPPORT STAFF AT ALL 
LEVELS TO BECOME INVOLVED BY... 


e Building capacity for public engagement with research through creating champions, networks, 
professional development, partnership working and creation of resources within 12 months 
e Establish practical support for public engagement with research e.g. central support unit within 


6 months 


e Training provision for public engagement with research to be developed and delivered within 


24 months 


e Communicate clear messages about the value and purpose of PE and how to get involved 
through events and conferences; publicity materials; newsletters; web presence ongoing 
e Provision of opportunities/platforms for researchers to engage with the public on an annual 


basis 


Potential Outputs 


Breadth and depth of participation Staff and students regularly use 


in activities. — e.g. numbers of staff 


trained and/or advised; attending work 
events; subscribing to newsletters Improved staff practices and 
or joining networks; champions / attitudes to public engagement 


coordinators created 


Professional development scheme 


Partnerships developed 
Resources created 

Public engagement activities 
completed 


Public engagement more 


meetings, networks, activities 


Increased demand for core public 


engagement services 
Greater confidence in public 
engagement in those 
participating in training or 
platforms for engagement 


Public engagement opportunities 


benefit the public in relevant 
ways 


the resources to inform their own 


regularly included in institutional 


Potential Outcomes Potential Impacts 


Clear understanding where to go to get 
support for public engagement 

Raised awareness of the value of 
engagement, with champions equipped 
to support their colleagues to get 
involved 


Gathering evidence 


Once you have developed your logic model you need to consider what you want to know about 
your programme. You may wish to focus on evaluating the results of the activity (‘summative’ 
evaluation), but don’t forget how useful evaluation can be when used ‘formatively’ to inform the 
development of your approach, or to provide ongoing reflection on what is working well, and 
where improvements could be made. 


Initially you need to consider the overall questions your evaluation will address. Questions might 

include: 

e Have the activities you have chosen to do led to the desired outcomes 

e What have been the most effective interventions for supporting public engagement within the 
research community, and why? 

e Have staff and students recognised a culture change in your institutions support for public 
engagement? 

e What types of intervention matter most to your staff and how can they be implemented 


It is great to start from these questions to inform your approach to evaluation. One of the primary 
audiences for this work will be you and your team, and therefore it is important to think through 
how evaluation will help you do your work well. It is also important to think through the evidence 
you may need to justify your business case. 


Having an external evaluator can really help. Some universities choose to have an evaluation 
officer as part of their engagement team, building capacity for effective evaluation, whilst 
providing a focal point to evaluation to the work of the team. Others bring in expert consultants to 
help inform their approach, and provide tools to enact it. 


Once you know your questions it is important to consider how you will approach gathering 
relevant data. Here are a few mechanisms commonly used to evaluate culture change and 
engagement support. 


Staff surveys, focus groups and interviews 


These are really effective tools if you want to find out whether people’s attitudes and approaches 
have changed over time. Clearly it is important to base line this at the beginning of your work, so 
that you can assess whether any change has happened. You can also do some of this work as part 
of events you run to support your engagement work. 


Culture change can take time, and it is important to consider how often you need to assess where 
things have got to. The EDGE tool provides a great focal point for this — and assessing progress on 
a biannual basis can provide useful intelligence as to how things are changing. 


Enquiry logs 


If you are working as a Public Engagement Professional a useful way of exploring how things 
change overtime is to capture enquiries and responses. This data can help you track if and how the 
nature of these enquiries change over time. For example, as awareness increases you would 


expect to have more people draw on your services; however, as you create effective online 
resources you would expect the nature of the questions people ask you to change. 


Other data sources 


Universities are consistently having to report back to funders about their work. Therefore you may 

find the existing data sets helpful. For example: 

e Data on awareness, support and capacity for public engagement within institutions is captured 
in two national surveys, the Careers in Research Online Survey (CROS) and the Principal 
Leaders and Research Leaders Survey (PIRLS). 

e RCUK quality assurance data is collected on a 6 yearly basis, and includes questions relating to 
public engagement 

e HEFCE require universities to provide data relating to their HEIF funding and questions linked 
to public and community engagement are included in the Higher Education Business and 


Community Interaction survey (HEBCIs) 
e REF 2021 returns and applications for research funding may provide useful data 


NCCPE resources and support 


The NCCPE offer a baseline survey resource to help you assess staff and student perceptions of the 
support the institution or department offer for public engagement against the EDGE tool. 


We also offer the Engage Watermark, a rigorous evaluation of the current public engagement 
support culture at your university or faculty which considers a range of sources of evidence 
including: documentary evidence; survey of staff, students, and partner organisations; a site visit; 
interviews and focus groups; and a review of future planning. Get in touch if you want to access 
this support. 


The NCCPE also run various evaluation courses, and can offer bespoke consultancy and training. 
Do get in touch if you could like advice or guidance about what we can offer. 
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